ChatGPT can miss even basic java concepts and make syntax errors

I started using ChaptGPT just recently and i have been noticing that still has a long way to go from replacing Developers. That being said still has huge potential and can be great tool if you are willing to do the work and make up the current short comings.

I have used it mostly to refactor my while creating a vulnerable web application that can be used to practice penetration testing. So far it has proven very useful but i can’t help to notice it is falling short in some instances where i expected to excell.

So one of the things I came across was it is possible that ChatGPT can miss basic java concept like superclass and when it is required by class that is extending the superclass to use the annotation @Override to override method.

Here is the superclass called “SocketController”

package com.enoch.testapp3.controller;

import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;

import java.io.IOException;
import java.net.Socket;

@RestController
@RequestMapping("/socket")
@CrossOrigin(origins = {"http://localhost:3000", "http://192.168.1.80:3000"}) // Adjust as needed
public class SocketController {

   @GetMapping("/connect")
   @ResponseBody
   public ResponseEntity<String> connectToSocket(@RequestParam("host") String host, @RequestParam("port") int port) {
       try {
             System.out.println("🔍 Attempting to connect to socket: " + host + ":" + port);

             Socket socket = new Socket(host, port);
             return ResponseEntity.ok("Socket connected to " + host + " on port " + port);
      } catch (IOException e) {
             System.err.println("❌ Failed to connect to socket: " + host + ":" + port);
             e.printStackTrace();
             return new ResponseEntity<>("Failed to connect to socket", HttpStatus.FORBIDDEN);
      }
   }
}

As you can see from the above source code the superclass doesn’t have method called applyFilter() declared. Now let us see what ChatGPT shared as new class that extends the superclass “SocketController”. The new class is called “UseSocketController with source code below.

package com.enoch.testapp3.controller;

public class UseSocketController extends SocketController {

@Override
public boolean applyFilter() {
    try {
        // Invoke the vulnerable method
        this.connectToSocket("127.0.0.1", 8080); // Use the IP and port of your choice
        return true; // Indicate that the filter was successfully applied
    } catch (Exception e) {
        e.printStackTrace();
        return false; // If there’s an error, return false
    }
}

}

As you can see from the source code above it is trying to override method which is basic java syntax error as the superclass doesn’t have the method “applyFilter()” declared and doesn’t require the annotation “@Override”.

I asked ChatGPT why we need to override the method and ChatGPT admitted it is mistake and corrected the code by removing the @Override annotation.

ChatGPT gets improved with every new model that tries to outperform the previous models but this is still a very surprising factor that it is making simple syntax errors like this. I believe ChatGPT still has a long way to go to even replace junior level developers in my opinion.

I have also shared the video on Youtube demonstrating the issue.

Share this post: on Twitter on Facebook