Stuck in having
You didn’t mention which pagination functions you use to output the paged links. I’m fairly sure they only work for main post loops. Once you make a custom query, the usual pagination functions don’t work because they still think you are working off the main post loop. They can seem to work because there is often some overlap in queries. Sooner or later they will fail on custom queries.
If you are not making use of the main post loop, your best option is to alter the main query to meet your needs through the “pre_get_posts” action. Of course this is not an option if your query is a secondary query after the main loop runs.