CS 4960/6960: Homework

Note: On this page, I will post a stream of HW problems. If you spot typos/clarifications, please ask! The same holds if you do not understand some notation.

1. Let $c\in {\mathbb{R}}^n$$c \in \mathbb{R}^n$ be some non-zero vector (i.e., at least one of its entries is non-zero). Suppose $x=(x_1,x_2,\dots ,x_n)$$x = (x_1, x_2, \dots, x_n)$ is a random vector obtained by setting each $x_i$$x_i$ to be $0$$0$ or $1$$1$ with probability (1/2), independently of the other vectors.

   (i) Prove that $Pr[⟨c,x⟩=0]\le \frac{1}{2}$$\Pr \left[ \langle c,x \rangle = 0 \right] \le \frac{1}{2}$. (Hint: focus on some index $i$$i$ where $c_i\ne 0$$c_i \ne 0$, now argue about the probability after conditioning on any setting of values for $x_j$$x_j$, for $j\ne i$$j \ne i$.)

   (ii) Give an example where equality occurs.

2. As we proved in the class the recurrence relation for the expected runtime of QuickSort is as below (and the base case is $T(1)=0$$T(1)=0$. Prove that the upper bound for $T(n)$$T(n)$ is $O(n\log (n))$$O(n \log(n))$.

   $T(n)=(n+1)+\frac{2}{n}(T(0)+T(1)+...T(n-1)).$$T(n) = (n+1) + \frac{2}{n}(T(0) + T(1) +. . .T(n-1)).$

   [Hint: Prove an intermediate step that $\frac{T(n)}{n+1}=\frac{T(n-1)}{n}+\frac{2}{n+1}$$\frac{T(n)}{n +1}=\frac{T(n-1)}{n} + \frac{2}{n+1}$]

3. The goal of this exercise is to show that non-negativity is required for Markov's inequality. Concretely, give an example of a random variable $X$$X$ such that $E[X]=1$$E[X] = 1$ and $Pr[X>4]=0.9$$\Pr[ X> 4] = 0.9$.

4. Suppose we take the numbers $1,2,...,n$$1, 2, ..., n$, and randomly permute them, to obtain the sequence ${\pi }_1,{\pi }_2,\dots ,{\pi }_n$$\pi_1, \pi_2, \dots, \pi_n$. Let $X$$X$ be the random variable that counts the number of "fixed" points, i.e., $j$$j$ such that ${\pi }_j=j$$\pi_j = j$. What is $\mathbb{E}[X]$$\mathbb{E}[X]$?
   [Hint: Define random variables $Y_j$$Y_j$ = $1$$1$ when ${\pi }_j=j$$\pi_j = j$ and $Y_j$$Y_j$ = $0$$0$ when ${\pi }_j\ne j$$\pi_j \neq j$​ and use linearity of expectation. ]

5. (a) Prove the following probability statement: let $X_1,X_2,\dots ,X_n$$X_1, X_2, \dots, X_n$ be independent random variables. Then, $\text{var}(X_1+X_2+\cdots +X_n)=\sum _{i=1}^n\text{var}(X_i)$$\text{var}(X_1 + X_2 + \dots + X_n) = \sum_{i=1}^n \text{var}(X_i)$ [Hint: recall that the variance of a random variable $X$$X$ with mean $\mu$$\mu$ is defined as $\mathbb{E}[(X-\mu {)}^2]$$\mathbb{E}[(X-\mu)^2]$.] (b) Show that for the above to hold, you don't need all the $X_i$$X_i$ to be independent, but you need only that $\mathbb{E}[X_i{X}_j]=\mathbb{E}[X_i]\mathbb{E}[X_j]$$\mathbb{E}[X_i X_j] = \mathbb{E}[X_i] \mathbb{E}[X_j]$ for all pairs $i,j$$i,j$. [If you're interested, you should ask yourself, doesn't this mean that $X_i,X_j$$X_i, X_j$ are independent? We will discuss this point in class on Monday, Jan 29.]

6. (Optional) A set of $n$$n$-bit binary strings $\mathcal{C}=\{s_1,s_2,\dots ,s_N\}$$\mathcal{C} = \{s_1, s_2, \dots, s_N\}$ is said to be an error correcting code (ECC) if for all $i,j\in [N]$$i, j \in [N]$ and $i\ne j$$i \ne j$, we have $d(s_i,s_j)\ge n/4$$d(s_i, s_j) \ge n/4$, where $d(s_i,s_j)$$d(s_i, s_j)$ denotes the Hamming distance, defined as the number of indices in which $s_i$$s_i$ and $s_j$$s_j$ differ. Background. In coding theory, one wishes to construct ECCs with $N$$N$ as large as possible (for a given $n$$n$). This is because we should think of $s_1,...,s_N$$s_1, ..., s_N$ as encodings of "messages" $\{1,2,...,N\}$$\{1,2, ..., N\}$, and we would like to encode as many messages as possible. Problem. Prove that there exists an ECC with $N>2^{0.1n}$$N > 2^{0.1n}$. Follow the random greedy addition algorithm, and argue that it produces a set $\mathcal{C}$$\mathcal{C}$ whose expected size is $\ge N$$\ge N$:

   start with C = emptyset
   for i = 1, 2, ..., 2N:
     set s_i = random n-bit string 
     if d(s_i, s_j) < n/4 for some j<i, discard s_i;
     else, add s_i to C
   return S
   

   [Hint: whenever $|\mathcal{C}|<2N$$|\mathcal{C}| < 2N$, show that the probability that an iteration of the for loop "succeeds" is at least $1/2$$1/2$. You may find it useful to assume that $(\genfrac{}{}{0ex}{}{n}{0})+(\genfrac{}{}{0ex}{}{n}{1})+\dots (\genfrac{}{}{0ex}{}{n}{n/4})<\frac{2^{0.9n}}{2}$$\binom{n}{0} + \binom{n}{1} +\dots \binom{n}{n/4} < \frac{2^{0.9n}}{2}$ ]

This completes HW 1. It is due on Wednesday, February 7 (11:59). If you need more time, please ask by email at least 1 day before the deadline.

7. The key property of convex functions over a convex domain (used in most optimization algorithms) is that a local minimum is also a global minimum. This condition can be stated in many ways, and here's one. Let $f$$f$ be a convex function, and let $D\subseteq {\mathbb{R}}^n$$D \subseteq \mathbb{R}^n$ be a convex set (which is the domain). Suppose $x\in D$$x \in D$ satisfies the property that for some $\rho >0$$\rho>0$, $f(y)\ge f(x)$$f(y) \ge f(x)$ for all $y\in \text{Ball}(x,\rho )$$y \in \text{Ball}(x, \rho)$. (In other words, $x$$x$ is a local minimum). Now prove that for all $z\in D$$z \in D$, we must have $f(z)\ge f(x)$$f(z) \ge f(x)$.

   (Hint. Recall that one definition of convexity is that for all $t\in [0,1]$$t\in [0,1]$ and points $x,y$$x, y$, $f(tx+(1-t)y)\le tf(x)+(1-t)f(y)$$f(tx + (1-t)y) \le tf(x) +(1-t) f(y)$. Use this to prove by contradiction. Suppose there exists $z$$z$ such that $f(z)<f(x)$$f(z) < f(x)$, and now use the definition of convexity to contradict the local minimum property. Also, in your proof, where are you using the convexity of $D$$D$?)

8. Another nice property of a convex function $f$$f$ over any bounded (and closed, to be precise) domain $D\subseteq {\mathbb{R}}^n$$D\subseteq \mathbb{R}^n$ is that the maximum value of $f$$f$ over $D$$D$ is always attained at a "boundary" point; more formally, there is always a point on the boundary that attains the maximum value over $D$$D$. Prove this fact.

   (Hint. Geometrically, if you draw a line through any point $x$$x$ that is not on the boundary, it will intersect the boundary in (at least) two points, one on either side.)

9. Linear classifiers (passing through the origin) are probably the most classic models in machine learning. Given a set of points $x_1,x_2,\dots ,x_m\in {\mathbb{R}}^n$$x_1, x_2, \dots, x_m \in \mathbb{R}^n$ and their labels $y_1,y_2,\dots ,y_m$$y_1, y_2, \dots, y_m$, the goal is to find a "hypothesis" that is a vector $w\in {\mathbb{R}}^n$$w \in \mathbb{R}^n$ such that $\text{sign}(⟨w,x_i⟩)=y_i$$\text{sign}(\langle w, x_i \rangle) = y_i$ for all $i\in [m]$$i\in [m]$.

   Solve the problem of finding such a $w$$w$ using a linear program. (You may assume that there exists such a $w$$w$.) (Hint. You will be able to express the problem just as a question of finding any feasible solution. There is a subtlety here about avoiding having $⟨w,x_i⟩=0$$\langle w, x_i \rangle = 0$. Think about how you can achieve this...)

10. Recall the LP relaxation for the set-cover problem, where we had $m$$m$ sets and $n$$n$ elements. As we did in class, suppose that the sets are called $S_1,S_2,\dots ,S_m$$S_1, S_2, \dots, S_m$ and the elements are $1,2,\dots ,n$$1, 2, \dots, n$. For every $j\in [n]$$j \in [n]$, the LP has the constraint: $\sum _{i\text{ such that }j\in S_i}{x}_i\ge 1.$$\sum_{i \text{ such that } j \in S_i} x_i \ge 1.$ Now suppose we know that for some parameter $D$$D$, every element $j$$j$ is contained in at most $D$$D$ sets. Show how to obtain a factor $D$$D$ approximation algorithm for set-cover on such instances. (In other words, give an algorithm that is guaranteed to output a solution no worse than factor $D$$D$ times the optimal.) (Hint. The solution is discussed in one of the lecture notes linked on the course page.)

11. Recall the linear program we saw in class for the facility location problem. Suppose the clients are numbered $1,2,\dots ,n$$1, 2, \dots, n$, and the facilities are numbered $1,2,\dots ,m$$1, 2, \dots, m$. The variables are (a) $y_1,y_2,\dots ,y_m$$y_1, y_2, \dots, y_m$, where $y_i$$y_i$ represents if $i$$i$ is picked or not, and (b) $x_{ji}$$x_{ji}$, that denote if client $j$$j$ is assigned to facility $i$$i$. In the LP relaxation, we imposed the constraints $0\le y_i\le 1$$0 \le y_i \le 1$ and $0\le x_{ji}\le 1$$0 \le x_{ji} \le 1$, for all $i,j$$i,j$. The objective had "opening cost", which was $\sum _{i\in [m]}{f}_i{y}_i$$\sum_{i \in [m]} f_i y_i$, and "connection cost", which was $\sum _{j\in [n]}\sum _{i\in [m]}d(j,i)x_{ji}$$\sum_{j \in [n]}\sum_{i \in [m]} d(j,i) x_{ji}$, where $d(j,i)$$d(j,i)$ is the distance from client $j$$j$ to facility $i$$i$.

    We also imposed the constraint $\sum _i{x}_{ji}=1$$\sum_i x_{ji} =1$ for all $i$$i$ (i.e., every client is assigned to some facility). Recall the definition $R_j=\sum _{i\in [m]}d(j,i)x_{ji}$$R_j = \sum_{i\in [m]} d(j,i) x_{ji}$, which is basically the connection cost of the LP solution for client $j$$j$. In class, we showed that every client $j$$j$, if $B_j$$B_j$ denotes the ball of radius $2R_j$$2R_j$ around client $j$$j$, then $\sum _{i\in B_j}{y}_i\ge \frac{1}{2}$$\sum_{i \in B_j} y_i \ge \frac{1}{2}$.

    Use this observation to analyze the following randomized rounding procedure: for all $i$$i$, open facility $i$$i$ with probability $min(1,\alpha y_i)$$\min(1, \alpha y_i)$, for some parameter $\alpha >1$$\alpha>1$.

    (a) Prove that for every client $j$$j$, the probability that we don't open any facility at distance $\le 2R_j$$\le 2R_j$ around it, is at most $\exp (-\alpha /2)$$\exp(-\alpha / 2)$. (b) [Extra credit, optional] Show how you would obtain an approximation algorithm for facility location using this idea, and write out the approximation factors for the opening cost and the connection costs separately.

This completes HW 2. It is due on Wednesday, March 13 (11:59). If you need more time, please ask by email at least 1 day before the deadline.