Don't worry. It'll be fine. People have wistful thoughts about kids sometimes - I have myself - but that's not the same as "OMG must breed!". He already knew you didn't want any more and he didn't go into this lightly - if that was going to be a breaking point it would have happened already, I think.