Your enthusiasm for the project definitely shows through in the document and your posts here, which is great!
Basis the rules off an existing system might make them more stable, but that's not nearly as fun for you as a designer
I think asking yourself a simple question will help you decide a lot:
1. Am I making this game for me and my friends to play, or at my hobby shop, or to market for money?
If you're making the game for you and your friends, great, do whatever you want that is the funnest
to you. If it happens to be fun to other people that's just a nice bonus.
This can also help you figure out questions like: "Do I need a
D20,
D12, and
D6?" Because if you're trying to market the game or have it appeal to other people, they might not have those dice on hand. Whereas if you're just playing it with friends, then you know full well if you have those dice around.
Same goes for table size. Personally I always start by basing my games off the table I'll be playing them on at home, instead of trying to set it for some "standard" size.
Annnnnnyway, enough philosophy. A few comments specifically on the rules:
1. I'd recommend reformatting the rulebook to two-columns. It's horribly tedious to do later, but is fairly standard.
2. An overview of what a wargame is could help. Just a basic paragraph explaining that you'll use a measuring tape to move models around a table, with dice to resolve things.
3. Instead of a Scatter Die (which can be a bit hard to get for non-
40k people, and I believe is technically proprietary) you could use your
D12 like a clock face (ie: a roll of 12 would scatter directly north), or some other scheme. Again this comes down to if you're just playing the game with friends or not.
4. Not sure if I'm sold on the format of the "Playing the game" section. Maybe it would work better as a visible flowchart diagram (with boxes and arrows and stuff) instead of a set of numbers? I feel somewhat like this section is more complicated than it needs to be. At the same time I think you're cramming way too much information in here. Seems like it could be compressed to: Activate, Move/Shoot, Assault, and then the details of those points described later in the document.
5. I'm not a fan of having to consult a chart/table of Weapon
Str vs Armor. It's cludgy in
40k, but you're writing your own system, so you could think up something funner

I also like systems where shooting is resolved in a single roll, instead of rolling to hit then rolling to wound.
6. Boost to me seems too random and game changing. For example the RGM-79SC
GM SNIPER has a 10" boost, but only a roll of 6 on a
D6. That's a HUGE swing of movement based on very low odds.
7. I like the idea of "Choosing how you want to play", but I'd consider putting this at the end of the document. When reading the rulebook I got to this section before I knew what half the stuff it's talking about even is. I'd also consider some kind of chart or sheet to track what rules are being used in the current game.
8. Weapon Accuracy also seems to have taken a nod from
40k. Have you played many other game systems besides
40k? Right now the Accuracy is exactly the same as
BS, just on a
D20. Why not just make Accuracy X+, instead of a number that you have to do math to figure out what you need to hit? So for example the ZAKU I has a 105mm Machine Gun with a Weapon Accuracy of 10, which works out to 11+. Why not just write 11+ on the stat sheet?
9. Technically your range bands overlap. Short is 6-18", but Medium is 18-30", so if a target is at 18" are they at Short or Medium? In general the whole range stuff seems confusing to me, with the "1 or more classes too short" applying a modifier and stuff. I think you need to take a step back and figure out what you're trying to achieve with ranges, and then work on rules that support that idea in a simple and easy to use way.
10. I like the idea of target position affecting accuracy, as it helps motivate players to move instead of just "camping" and sitting still.
11. You have all these extra boosting actions, like Evasive Maneuver, but they aren't mentioned earlier in your flowchart. How do you foresee players tracking if they've used their boost action or not? Since to use Evasive (which is used in the enemy turn) you'd have to know you didn't boost during your previous turn.
12. Roll for priority...why are you using
D6 here (with such a small range of numbers) when you have
D12s and
D20s available? Why not use
D20 to have a greater chance to avoid ties between players?
13. Assault after moving/shooting. To me this (again) seems like a
40k hold over. Why not just have a unit being able to move/shoot/assault (aka "Perform an Action") in any order, instead of having a special phase for it?
14. You have some special rules (
USRs for lack of a better term), but then some weapons (like the Bazooka) have their own custom rules buried in the text. Why not push the weapon rules to this
USR chart? The same goes for a lot of unit specific weapons, like the "Cracker Grenade".
15. Border Skirmish scenario goes back to the Boost problem I was talking about, where if one player rolls their Boost really well they could win by getting across the table, just from a few lucky dice rolls. I imagine you want more player input than that?
16. I'm not sure why Shield Checks are put in their own little box, whereas other mechanics are just loosely explained in the flowchart? Does using a Shield have a downside? If not why wouldn't a player always roll for it?
17. Do players write down their current Integrity on their stat sheet? Or do they track it on the table with markers/counters/tokens? If the former I'd increase the space of the Integrity box so players can easily write in there. Or I'd even do little circles they can fill in (like 4 circles for 4 Integrity) or checkboxes or something, to make the process as painless as possible.
Note: I didn't go through the advanced rules or the campaign stuff.
18. I'd also consider updating the stat sheet to have the stats follow the process of a turn. What I mean is Movement is (probably) going to be first, so it should be the leftmost item. Integrity happens after an attack, in the enemy turn, so it's kind of last, and should be the rightmost column.
Most of that is just formatting and clarification. Some of the issues do make it a bit harder for a reader to understand how the game flows, so I might not have the full picture.
But I guess my biggest concern is talking through a typical attack. I mean attacks are the most common, so you're going to be doing this process a ton. To me it reads as: "Okay I'm shooting with Weapon Accuracy 14, *measures and does some wacky range calculation* at a bad range, so -5, so 9 accuracy now, but it's at your back, so +3, so 12. Oh but wait you want to Evade? *enemy uses boost action to evade, rolls
D6*. Okay you evaded, so that reduces my Accuracy by -4 to 8. Now *looks at chart* that means I need 13+ *rolls to hit* okay I hit. You have 5 armor, but you're in light cover, so +1, so 6. Do you want to Shield? You do? *opponent rolls shield* and you made your shield, so that's plus what? *enemy looks at sheet* +1, so 7 armor now. Okay my Weapon Strength is 9, so *checks chart* so I need 5+ on my
D12. So you're down 1 integrity *opponent updates their sheet with the new integrity*".
So 5 calculations, 2 chart lookups, and 4 dice rolls (between you and your opponent), 1 sheet update.
Streamlining this process, while still maintaining the flavor and options you want, is going to be a tough challenge. But then again figuring out fun, fluid, well designed core resolution mechanics always is.