Using ajax on categories and wordpress loops

Yes this is possible with WordPress but I would not use index.php but a custom front-page.php template then create a page called home and set it as the front page in options-general.

For your category menu:

<?php

$categories = get_categories(); ?>

<ul id="category-menu">
    <?php foreach ( $categories as $cat ) { ?>
    <li id="cat-<?php echo $cat->term_id; ?>"><a class="<?php echo $cat->slug; ?> ajax" onclick="cat_ajax_get('<?php echo $cat->term_id; ?>');" href="#"><?php echo $cat->name; ?></a></li>

    <?php } ?>
</ul>

Where calling our jQuery ajax function when a category is clicked and passing the cat ID to jQuery in the name of the function.

The html div place holder where your posts will load via ajax:

<div id="loading-animation" style="display: none;"><img src="https://wordpress.stackexchange.com/questions/35849/<?php echo admin_url ("images/loading-publish.gif' ); ?>"/></div>
<div id="category-post-content"></div>

The jQuery function called via the onclick handler in the menu items:

<script>
function cat_ajax_get(catID) {
    jQuery("a.ajax").removeClass("current");
    jQuery("a.ajax").addClass("current"); //adds class current to the category menu item being displayed so you can style it with css
    jQuery("#loading-animation").show();
    var ajaxurl="<?php echo admin_url( "admin-ajax.php' ); //must echo it ?>';
    jQuery.ajax({
        type: 'POST',
        url: ajaxurl,
        data: {"action": "load-filter", cat: catID },
        success: function(response) {
            jQuery("#category-post-content").html(response);
            jQuery("#loading-animation").hide();
            return false;
        }
    });
}
</script>

The WordPress PHP function to return the posts from the category selected.

add_action( 'wp_ajax_nopriv_load-filter', 'prefix_load_cat_posts' );
add_action( 'wp_ajax_load-filter', 'prefix_load_cat_posts' );
function prefix_load_cat_posts () {
    $cat_id = $_POST[ 'cat' ];
         $args = array (
        'cat' => $cat_id,
        'posts_per_page' => 10,
        'order' => 'DESC'

    );

    $posts = get_posts( $args );

    ob_start ();

    foreach ( $posts as $post ) {
    setup_postdata( $post ); ?>

    <div id="post-<?php echo $post->ID; ?> <?php post_class(); ?>">
        <h1 class="posttitle"><a href="https://wordpress.stackexchange.com/questions/35849/<?php the_permalink(); ?>"><?php the_title(); ?></a></h1>

        <div id="post-content">
        <?php the_excerpt(); ?>
        </div>

   </div>

   <?php } wp_reset_postdata();

   $response = ob_get_contents();
   ob_end_clean();

   echo $response;
   die(1);
   }

We are using output buffering which helps prevent the function from failing which can happen sometimes with WordPress Ajax.