Gaussian-RBM with NRLU hidden units (in DBN)?

I'm working on a RBM (for a DBN) for image classification. I' working with two RBM layers. The first has Gaussian visible units and binary hidden units and the second has binary visible units and softmax hidden units. It works quite well. I now want to try to use Noise Rectified Linear Unit as the hidden layer, but I fail to understand how to implement them. Everything I've tried has just led to terrible results.

Now, if I understand correctly: The activation probability of a ReLU is simply p = max(0, x + N(0, 1)), but then how do I sample the values to activate the visible units ? Should the noise only be used in sampling and not in activation probabilities ?

Another thing: in some papers I saw that the noise is to be N(0,1) and some others use N(0,sigmoid(x)).

So, what should be the activation function and how the values should be sampled ?

Answers


Apparently:

Using max(0, x) as activation probability function and using max(0, x + N(0, Sigmoid(x)) for sampling seems to work for the RBM.


Need Your Help

J2ME Midlet exception :IOException reading reader invalid first byte

java java-me midlet midp

When i run my midlet with Java Wireless toolkit, midlet runs correctly, but when it try to parse a textfield, following error occurs;

What should I watch to debug this Core-Data / NSFetchedResultsController bug?

iphone uitableview core-data nsfetchedresultscontrolle

I've got a bug in my Core Data implementation that I cannot seem to get a hold of, and I need some tips on where I should be looking to figure out how to fix it.

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.