“no function to return from, jumping to top level”

CheckForMersennePrimes has extra {} on line 2 and line 4 (inside for loop) of this function, like vec<-MarsenneNumber(x){if()...}

I removed those extra braces.

CheckForMersennePrimes <- function( x ){
  b <- length( MarsenneNumber( x ) )
  for ( i in 1:b ){
    vec <- MarsenneNumber( x )
    if ( factorlist( vec[i] ) != 1 ){
      vec <- c( -vec[i] )
    }
  }
  return( vec )
}

I think, your intention is to get values of vec in which factorlist(vec[i]) is equal to 1. If this is true, then you have to initiate vec outside the for loop and check for this condition. Then finally return a1 which contains all vec values passing the if condition.

I also shortened your code by removing b variable. You do not need it. Instead you could do seq_along(vec). It will do the same thing without b variable.

CheckForMersennePrimes <- function( x ){
  vec <- MarsenneNumber( x )
  a1 <- c()
  for ( i in seq_along( vec ) ){
    if ( factorlist( vec[i] ) == 1 ){
      a1 <- c( a1, vec[i ] )
    }
  }
  return( a1 )
}

Further condensing the code will give same output. I removed seq_along and indexing of vec

CheckForMersennePrimes <- function( x ){
  vec <- MarsenneNumber( x )
  a1 <- c()
  for ( i in vec ){
    if ( factorlist( i ) == 1 ) {
      a1 <- c( a1, i )
    }
  }
  return( a1 )
}