No, the Time Tracker users cannot make any changes related to the objects that are set up for tracking time in your org. 

Only a Salesforce administrator can make any changes related to the Time Tracker for Salesforce app in Salesforce.

The Salesforce Admin can change the objects for tracking time, mark them as Inactive or delete them in Salesforce.