The publish date is either when the post is originally published, or the date that you choose to set it as published in the Editor, so that is the part you can manually control. The modified date is whenever an update has most recently occurred, such as when the Update button is pressed in the Editor.
You might end up wanting to add a custom field to hold a “public date.” The downside there is it would be stored in postmeta, making your queries less performant. Another alternative would be adding custom code that adds a date control in the Editor and forcibly sets the modified date to whatever you are choosing manually.