a Sic ,@sdZddlZddlmmZddlmZddl m Z ddl m Z ddl m Z ddgZe d Gd d d e jZd d ZddZdS)zGKeras layers that implement explicit (approximate) kernel feature maps.N) initializers) base_layer) input_spec) keras_exportgaussian laplacianz/keras.layers.experimental.RandomFourierFeaturescsJeZdZdZdfdd ZfddZd d Zd d Zfd dZZ S)RandomFourierFeaturesa7Layer that projects its inputs into a random feature space. This layer implements a mapping from input space to a space with `output_dim` dimensions, which approximates shift-invariant kernels. A kernel function `K(x, y)` is shift-invariant if `K(x, y) == k(x - y)` for some function `k`. Many popular Radial Basis Functions (RBF), including Gaussian and Laplacian kernels, are shift-invariant. The implementation of this layer is based on the following paper: ["Random Features for Large-Scale Kernel Machines"]( https://people.eecs.berkeley.edu/~brecht/papers/07.rah.rec.nips.pdf) by Ali Rahimi and Ben Recht. The distribution from which the parameters of the random features map (layer) are sampled determines which shift-invariant kernel the layer approximates (see paper for more details). You can use the distribution of your choice. The layer supports out-of-the-box approximations of the following two RBF kernels: - Gaussian: `K(x, y) == exp(- square(x - y) / (2 * square(scale)))` - Laplacian: `K(x, y) = exp(-abs(x - y) / scale))` **Note:** Unlike what is described in the paper and unlike what is used in the Scikit-Learn implementation, the output of this layer does not apply the `sqrt(2 / D)` normalization factor. **Usage:** Typically, this layer is used to "kernelize" linear models by applying a non-linear transformation (this layer) to the input features and then training a linear model on top of the transformed features. Depending on the loss function of the linear model, the composition of this layer and the linear model results to models that are equivalent (up to approximation) to kernel SVMs (for hinge loss), kernel logistic regression (for logistic loss), kernel linear regression (for squared loss), etc. Examples: A kernel multinomial logistic regression model with Gaussian kernel for MNIST: ```python model = keras.Sequential([ keras.Input(shape=(784,)), RandomFourierFeatures( output_dim=4096, scale=10., kernel_initializer='gaussian'), layers.Dense(units=10, activation='softmax'), ]) model.compile( optimizer='adam', loss='categorical_crossentropy', metrics=['categorical_accuracy'] ) ``` A quasi-SVM classifier for MNIST: ```python model = keras.Sequential([ keras.Input(shape=(784,)), RandomFourierFeatures( output_dim=4096, scale=10., kernel_initializer='gaussian'), layers.Dense(units=10), ]) model.compile( optimizer='adam', loss='hinge', metrics=['categorical_accuracy'] ) ``` To use another kernel, just replace the layer creation line with: ```python random_features_layer = RandomFourierFeatures( output_dim=500, kernel_initializer=, scale=..., ...) ``` Args: output_dim: Positive integer, the dimension of the layer's output, i.e., the number of random features used to approximate the kernel. kernel_initializer: Determines the distribution of the parameters of the random features map (and therefore the kernel approximated by the layer). It can be either a string identifier or a Keras `Initializer` instance. Currently only 'gaussian' and 'laplacian' are supported string identifiers (case insensitive). Note that the kernel matrix is not trainable. scale: For Gaussian and Laplacian kernels, this corresponds to a scaling factor of the corresponding kernel approximated by the layer (see concrete definitions above). When provided, it should be a positive float. If None, a default value is used: if the kernel initializer is set to "gaussian", `scale` defaults to `sqrt(input_dim / 2)`, otherwise, it defaults to 1.0. Both the approximation error of the kernel and the classification quality are sensitive to this parameter. If `trainable` is set to `True`, this parameter is learned end-to-end during training and the provided value serves as the initial value. **Note:** When features from this layer are fed to a linear model, by making `scale` trainable, the resulting optimization problem is no longer convex (even if the loss function used by the linear model is convex). trainable: Whether the scaling parameter of the layer should be trainable. Defaults to `False`. name: String, name to use for this layer. rNFc s|dkrtd|t|tr@|tvr@td|dt|dur^|dkr^td|tjf||d|||_||_||_ dS)Nrz5`output_dim` should be a positive integer. Received: z"Unsupported `kernel_initializer`: z Expected one of: z=When provided, `scale` should be a positive float. Received: ) trainablename) ValueError isinstancestrlower_SUPPORTED_RBF_KERNEL_TYPESsuper__init__ output_dimkernel_initializerscale)selfrrrr r kwargs __class__S/var/www/html/django/DPS/env/lib/python3.9/site-packages/keras/layers/kernelized.pyrs.   zRandomFourierFeatures.__init__cst|}|jdkr*td|jd||jdjdurHtd|tjdd|jdjid|_|jdj}t |j ||j fd}|j d||j ftj |d d |_|j d |j ftj tjd dtjd d d |_|jdurt|j ||_|j ddtj tjj|jddd|_t|dS)NzCThe rank of the input tensor should be 2. Received input with rank z% instead. Full input shape received: cThe last dimension of the input tensor should be defined. Found `None`. Full input shape received: )ndimaxes)shapeunscaled_kernelF)r r!dtype initializerr biasr )minvalmaxval kernel_scale)rTNonNeg)r r!r#r$r constraint)tf TensorShaperankr ndimsdimsvaluer InputSpec _get_random_features_initializerrr add_weightfloat32r"r RandomUniformnppir%r_get_default_scalecompatv1constant_initializerr(rbuild)r input_shape input_dimrrrrr<sb      zRandomFourierFeatures.buildcCsVtj||jd}t|tj}d|j|j}tj||d}tj ||j }t |S)N)r#?)ab) r+convert_to_tensorr#castr4r(r"matmulnnbias_addr%cos)rinputskerneloutputsrrrcalls zRandomFourierFeatures.callcCsFt|}|d}|jdjdur2td||dd|jS)Nrr)r+r, with_rankr/r0r concatenater)rr=rrrcompute_output_shapes  z*RandomFourierFeatures.compute_output_shapecsP|j}t|tst|}|j||jd}t}t t | t | S)N)rrr) rr rr serializerrr get_configdictlistitems)rrconfig base_configrrrrQs   z RandomFourierFeatures.get_config)rNFN) __name__ __module__ __qualname____doc__rr<rKrOrQ __classcell__rrrrrsq 6 rcCsjdd}|}t|trf|dkr0tjdd}n6|dkrRt|dd|d}ntd |d t|S) z/Returns Initializer object for random features.cSs.tjjdd|d}||ttj|dS)Nr r?)lowhighsizeg?)r6randomuniformtanr7)locrr!probsrrr_get_cauchy_samplessz=_get_random_features_initializer.._get_cauchy_samplesrr?)stddevrr )rbrr!z#Unsupported `kernel_initializer`: "z" Expected one of: )r rrr RandomNormalConstantr r)r$r!rdZrandom_features_initializerrrrr2s    r2cCs(t|tr$|dkr$t|dSdS)Nrg@r?)r rrr6sqrt)r$r>rrrr8sr8)rZnumpyr6tensorflow.compat.v2r9v2r+kerasr keras.enginerr tensorflow.python.util.tf_exportrrLayerrr2r8rrrrs    c