You can have problems only if database schema changed from older to new version.
To know if the your is the case, you can compare the file wp-admin/includes/schema.php for both the versions, see that online here for 3.8.1 and here for 3.5.1.
Note that a db index change should not beak the site, but it can cause performance issues.
If you think the schemas are compatible, before putting the old files back, I suggest you to also change the option in database pointing to current database schema.
In options table, there’s a row with option_name: db_version and option_value that contains the current database schema number, so I suggest to change manually it (using phpmyadmin or similar) to the old database version number. To know it, look ad wp-includes/version.php file of the older version. As you can see here, for WP 3.5.1 the version number was 22441 (is 26691 in WP 3.8.1).
As additional suggestion, be sure to have a complete database backup before doing anything.