Java http requests get banned on site

I am trying to get information from a site using a Java program I wrote.

When I am using selenium everything is ok, but when I try to use prowser or httpurlconnection the site blocks my IP.

I am setting user agent to Chrome/44.0.2403.155, and other parameters in the original site request (I get them with help of HTTPFox).

I am handling cookies (cookies I get from FirefoxWebDriver), but the result is always the same: my IP is blocked.

I get the information I need six times, and then site blocks my IP. When I don't use cookies ("logged out mode") everything is ok.

Answers


Is the site you accessing is over HTTPS or HTTP ?

Please try the code which will print you the GET request to the site.

public class ConnectToHTTPS {
    public static void main(String[] args) throws Exception {
        // Create a trust manager that does not validate certificate chains
        TrustManager[] trustAllCerts = new TrustManager[] {new X509TrustManager() {
                public java.security.cert.X509Certificate[] getAcceptedIssuers() {
                    return null;
                }
                public void checkClientTrusted(X509Certificate[] certs, String authType) {
                }
                public void checkServerTrusted(X509Certificate[] certs, String authType) {
                }
            }
        };

        // Install the all-trusting trust manager
        SSLContext sc = SSLContext.getInstance("SSL");
        sc.init(null, trustAllCerts, new java.security.SecureRandom());
        HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());

        // Create all-trusting host name verifier
        HostnameVerifier allHostsValid = new HostnameVerifier() {
            public boolean verify(String hostname, SSLSession session) {
                return true;
            }
        };
        // Install the all-trusting host verifier
        HttpsURLConnection.setDefaultHostnameVerifier(allHostsValid);


        URL url = new URL("YOUR SITE ADDRESS");
        HttpURLConnection cox= (HttpURLConnection) url.openConnection();      

       // URLConnection con = url.openConnection();

        Reader reader = new InputStreamReader(cox.getInputStream());
        while (true) {
            int ch = reader.read();
            if (ch==-1) {
                break;
            }
            System.out.print((char)ch);
        }


    }
}

OR

Simply try with setting Proxy in your java Program

    System.setProperty("http.proxyHost", "YOUR PROXY");

          URL url = new URL("YOUR WebSite");

            URLConnection con = url.openConnection();

            BufferedReader in = new BufferedReader(new InputStreamReader(
                    con.getInputStream()));

            String inputLine;
            while ((inputLine = in.readLine()) != null)
                System.out.println(inputLine);

            in.close();

    }

OR if Cert issue is solved

You can use GET once the above worked 

 // Install the all-trusting host verifier
            HttpsURLConnection.setDefaultHostnameVerifier(allHostsValid);

            String url = "http://www.google.com/search?q=mkyong";

            URL obj = new URL(url);
            HttpURLConnection con = (HttpURLConnection) obj.openConnection();

            // optional default is GET
            con.setRequestMethod("GET");

Need Your Help

MVC 4 Strongly Typed Data to View without ORM

asp.net-mvc linq entity-framework

I've seen other questions regarding using straight SQL for MVC data access, for example here. Most responses don't answer the question but ask

Using SetTimeOut to fadeout a list of element tiling on each other

javascript jquery html css

Suppose that I have a list of elements tiling on each other using z-index. For example:

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.