Before giving you an answer, I'd strongly, STRONGLY, recommend coming to grips with the basic rules here:
https://whc-cdn.games-workshop.com/wp-content/uploads/2019/01/Warhammer-40k-Battle-Primer-English-3.pdf
House rules are awesome, but the base rules need to be known so that massive confusion doesn't result when you encounter other players.
With that out of the way, here's the rules that are important:
#1 - Nothing does spill-over damage. If you deal more damage to a model than it takes to kill it, it just dies and the rest of the damage is wasted.
#2 - Mortal Wounds are dealt 1 at a time, so each of them can be considered to be 1 damage. In this way, if you roll to deal d3 mortal wounds to a unit of 1-wound models, and you roll a 3, you'll still kill 3 guys.
#3 - When you make a lot of attacks at once, because you're shooting a whole bunch of bolters, or slashing with a whole bunch of chainswords, we tend to total up and roll all the dice together. This is called "fast rolling" because you're going "fast" through rolling all the dice, but the actual rules are that you roll attacks 1 attack at a time (even if a gun shoots multiple shots, each of those shots is taken in sequence). Since all the attacks are identical and follow identical rules, you can roll all the dice together though.
This "fast rolling" can make it look like damage is spilling over to other models in a squad, but really, it's just that you've skipped going over 1 roll and 1 dead model at a time for the sake of expediency and because rolling all together or 1 at a time is exactly the same on a statistical basis.